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A  Human  •  C  entered  Engineering 


•  Goal:  effective  collaboration 

•  Collaboration  at  a  distance — enabled 
by  network  connectivity —  is  central  to 
21st  century  Command  and  Control 

•  Collaboration  technology  capabilities 
have  outpaced  understanding  of 
collaboration 

•  Need  insight  into  the  cognitive 
processes  involved  in  effective 
collaborative  in  order  to  best  design 
and  use  the  technology 

•  Our  focus:  collaborative  critical 
thinking. 

•  Our  objectives: 

•  Define, 

•  Measure,  and 

•  Strengthen  CCT  w/  tools  and  training 
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Template 

•  Team  Member  A:  States  assessment 

•  Team  Member  B:  Proposes  alternative  assessment 

Transcript: 

•  Mike:  Study  Thomas's  use  of  the  US  Calvery  with 
repeating  firearms  in  the  battle  of  Nashville.  Especially  the 
follow-up  where  Hood's  army  was  totally  destroyed  as  a  fighting 
force. 

•  Robert:  As  for  the  Battle  of  Nashville,  a  much  better  point  is 
don't  let  your  commander  become  a  opium  addict.  Much 
of  what  was  left  of  the  Army  of  Tennessee  had  already  been 
squandered  on  the  useless  assualts  at  Franklin.  While  Thomas 
did  a  commendable  job  of  defeating  the  remains,  the  issue  was 
hardly  in  doubt. 


Source:  sci. military  newsgroup 
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Template 

•  Team  Member  A:  Monitors  for  risk 

•  Team  Member  B:  I  dentifies  source  of  risk 

•  Team  Member  A:  Concurs 

Transcript: 

•  Harriet:  1 1  looks  like  one  subject  was  sort  of  pulling  it  up 
there... 

•  Sam:  Yeah,  there's  one  slow  subject,  uh  or  two  at  the  end. 

•  Keith:  Well  you  could  have  a  selection  effect.  I  n  that . . . 

•  Sam:  ...  in  the  examples... 

•  Keith:  Slower  subjects  in  the  example  condition,  because 
they  fail. 

•  Sam:  That's  correct,  uh  yeah. 


Source:  Chris  Schunn,  Ph.D.,  LRDC 
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Examples 


Template 

•  Team  Member  A: 

•  Team  Member  B: 

•  Team  Member  B: 

•  Team  Member  A: 

•  Team  Member  B: 


Monitors  for  risk 
I  dentifies  source  of  risk 
Plans  investigation  of  source  of  risk 
Prompts  for  contingency  plan 
Proposes  contingency  plan 


Transcript: 

•  Mike:  We  need  to  redirect  our  friendlies  to  account  for  SAM  A34's  relocation. 

•  Gavan:  Ok.  I  thought  this  SAM  was  fortified,  stationary. 

•  Mike:  Negative.  Coml  nt  has  just  reported  that  the  SAM  is  moving. 

•  Gavan:  I  see  it.  Didn't  ELI  NT  and  I  Ml  NT  report  no  movement  and  no  support?  Doesn't 
COMMI  NT  get  their  information  from  the  other  two? 

•  Mike:  That's  my  understanding,  but  I  will  confirm  that. 

•  Gavan:  So,  we  should  check  back  to  make  certain  these  reports  are  correct.  Why 

don't  you  check  back  with  I  Ml  NT  and  I  'll  check  back  with  ELI  NT  to  verify  this  information. 

We  still  have  a  bit  of  time.  Ask  them  how  conclusive  their  information  is.  How  did  they 
decide  this  SAM  would  not  move? 

•  Mike:  Shouldn't  we  decide  on  a  time  to  abort  the  mission  or  at  least  to  make  a  final 
call? 

•  Gavan:  Yes.  Probably  the  safest  thing  to  do  would  be  to  cancel  the  mission  if  we  aren't 
certain  about  that  SAM.  We  can't  reroute.  Let's  huddle  no  later  than  0500  and  make  a 
final  call  no  later  than  0600. 


•  Source:  Aptima  intelligence  scenario 
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What  is  Collaborative  Critical  Thinking? 


•  Collaborative  Critical  Thinking  (CCT)  is  an  interactive  process  for 
evaluating  and  refining  assessments,  plans,  and  teamwork 
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Collaborative  Critical  Thinking 

I  nteractive  -  CCT  is  collaborative 
Evaluative  -  CCT  involves: 

•  Monitoring  &  perception  for  risk  or  uncertainty 

•  Assessing  the  priority  of  addressing  this  risk  (given  other  tasks) 

•  Identifying  source(s)  of  risk  or  uncertainty  (assumptions,  gaps,  conflicts) 
Productive  -  Leads  to  action  that  bear  on: 

•  problem  assessment . . .  reframes  the  problem 

•  problem  solutions 

•  Gathering  information  by  probing,  testing,  waiting 

•  Eliminating  the  problem  (e.g.,  an  enemy  outpost) 

•  Developing  contingency  plans 

•  team  state,  process,  and  structure 

•  team  CCT  skills 

Distributed  -  CCT  may  be  conducted  by  teams  distributed  over  space  or  time. 
Critical  factors: 

•  Conduits  for  communication  and  collaboration 

•  Transactive  memory  (of  who  said  what,  did  what,  can  do  what) 

•  Public  representations  of  problem  state  and  (transaction)  history 
Addresses  ill-defined  problems  -  No  agreed  upon  method  or  answer 
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Motivation  &  Objectives 

Collaborative  Critical  Thinking  (CCT)  defined 

Conceptual  Model 

Experiments 

Planned  CCT  Technology  and  Training  Products 
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•  Nature  of  mission 

/ 

•  Division  of  resources  & 
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responsibilities 

•  Distribution  of  expertise 
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Factors  Affecting  Ability 
to  Collaborate 

•Technology  Factors 
•Process/Skill  Factors 
•Team  Composition  Factors 


Tool 


Training 
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Factors  Generating  Need  to  Collaborate 

•Nature  of  mission 

•Division  of  resources  &  responsibilities 
•Distribution  of  expertise 

•Mission  is  complex  and  time  sensitive 

•Resources  &  responsibilities  are  divided  among  team  members 
•Expertise  is  distributed  between  team  members 
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Factors  Affecting  Ability  to  Collaborate 


•Technology  Factors 

•Process/Skill  Factors 
•Team  Composition  Factors 


Collaboration 

technology 

measures 

Reach:  Team 
connectivity 

Interconnectivity  of  team  members  to  each 
other  (“Communities  of  interest”) 

Reach:  Information 
connectivity 

Interconnectivity  of  team  members  to 
information  sources  (“Information  access”) 

Reach:  Translation 

Degree  to  which  tool  supports  translation 
between  representations,  domains,  and 
languages 

Richness: 

Structured  problem 
representation 

Degree  to  which  the  tool  provides  structured 
representations  of  the  problem  at  hand 

Richness: 

Deconfliction 

Degree  to  which  the  tool  supports  coordination 
of  activities  via  a  shared  workspace 
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Variables  We  Manipulate  -  Training 


Factors  Affecting  Ability  to  Collaborate 

•Technology  Factors 

•Process/Skill  Factors 

•Team  Composition  Factors 

CCT 

process 

measures 

Planning 
for  TC2T 

Collaboration  communications  (e.g.,  paraphrasing 
others,  explicit  statement  of  monitoring  plans  & 
critieria)  during  planning  stage. 

Monitoring 

Number  of  critiques  initiated  concerning  high 

priority  issues 

Diagnosis 

Number  of  gaps,  conflicts,  and  untested 
assumptions  identified 

Action 

Instances  of  probing  own  resources  for  data, 
testing  enemy  or  environment  for  data, 
intentionally  waiting  out  problem 
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Measures  We  Take 


Products  of  Collaboration 

■Solutions  and  plans 


•Shared 

knowledge  and  awareness 

Shared 

awareness 

measures 

Shared  situation 

awareness 

Degree  to  which  team  members  share  memory  for 
current  location  of  objects  in  the  tactical  picture 

Shared  predictions 
of  future  situation 

Degree  to  which  team  members  share  predictions  of 
the  location  of  objects  in  the  tactical  picture 

Shared  situation 

assessment 

Degree  to  which  team  members  share  assessments  of 
the  intent  of  entities  of  operational  interest 

Mutual  awareness 
of  goals 

Degree  to  which  team  members  express  shared  goals 

Mutual  awareness 
of  information 
needs 

Degree  to  which  team  members  anticipate  the 
information  needs  of  teammates 

Mutual  awareness 
of  next  action(s) 

Degree  to  which  team  members  anticipate  the  actions 

of  teammates 

Mutual  awareness 
of  workload 

Accuracy  with  which  team  members  estimate  the 

subjective  workload  of  teammates 
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Effects  of  Collaboration  on  Mission  Performance 

•  MOEs 


Synchronization: 

Proportion  of  tasks  executed  without  conflicts  in 

Allocation 

resource  allocation 

Synchronization: 

Execution 

Proportion  of  tasks  requiring  coordination  that  are 
successfully  executed 

Synchronization: 

Precision 

Lag  in  readiness  for  execution  of  synchronized 
events  between  first  and  last  operator  ready  to  act. 

MOEs 

Effects:  Enemy 
losses 

Proportion  of  targets  or  threats  destroyed 

Effects:  Friendly 
losses 

Proportion  of  friendly  forces  destroyed  or  lost 

Effects:  Delay  in 
enemy  ops 

Latency  in  enemy  maneuvers  or  actions  (e.g., 
bombing  targets)  due  to  friendly  actions 

Effects:  Team 

Average  delay  in  task  execution  relative  to  optimal 

process 

task  execution  schedule 
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Hypotheses 

•  HI:  CCT  tools  and/or  training  improve  shared  awareness  of  uncertainty  and  risk 

(Ability/  Process) 

•  H21:  CCT  tools  and/or  training  increase  the  incidence  of  CCT  behaviors. 

(Ability/ Process) 

•  H3:  CCT  tools  and/or  training  improve  the  team  plans.  (Products) 

•  H4:  CCT  tools  and  training  improve  mission  execution  and  outcomes.  (Effects) 
Materials:  Military  scenario  in  which 

•  some  aspects  of  the  situation  are  well  defined,  others  are  not. 

•  some  risks  can  be  reduced  by  information  gathering  or  probing 

•  some  risks  cannot  be  reduced  and  require  contingency  plans 
Testbed:  Distributed  Dynamic  Decision-making  (DDD)  Simulation 

•  Team  research  testbed 

•  Collaboration  measurement  capability 

•  Developed  at  U.Conn,  freely  available,  used  at  25  labs 
Subjects:  ROTC  and  undergraduate  students 
Method: 

•  Pretest  domain  knowledge  &  critical  thinking  ability 

•  Scenario  (re) planning  phase  execution  phase 

•  Real  time  measures  of  CCT 

•  Posttest  measures  of  CCT 

Analysis:  Multi  level  modeling  supports  analysis  of  group. 
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individual,  their  interaction  (individual  on  this  team),  and  error  for  group  and  individual 
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•  Hypothesis: 

•  Team  awareness  of  deficits  in  shared  awareness 
(regarding,  e.g.,  risk  involved  and  predicted 
outcomes)  enable  teams  to  improve  awareness  and 
mission  effects. 

•  Tool  will  help  the  team  evaluate  its  state  and  identify 
opportunities  for  critical  thinking 

•  Each  team  member  will  answer  questions  about  the 
risk  involved  and  the  predicted  outcomes 

•  These  values  will  be  combined  to  calculate  the  team's 
mutual  awareness  of  risk  and  predicted  outcomes 
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Hypothesis:  CCT  training  will  help  teams  to: 

•  I  dentify  the  triggers  in  the  situation  that  require  CCT 

•  Weigh  benefits  of  engaging  in  CCT  across  all  priorities 

•  Exercise  their  CCT  faculties  to  improve  plans 
Possible  training  topics: 

•  Domain- independent  CCT 

•  Training  in  devil's  advocate  strategies 

•  Training  in  appropriate  situation  for  CCT  (e.g.,  need  for  high  quality, 
with  relatively  little  time  constraints) 

•  Domain- specific  CCT  -  Train  to  identify  and  critically  evaluate 
problem.  E.g.,  determine  why  there  are  problems  localizing 
enemy.  I  s  it  caused  by: 

•  deliberate  deception  by  enemy 

•  Reliability  of  source 

•  Stale  data 
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Project  Timeline 


28  August  2002  through  27  August  2004 


Schedule 

Task 

Type 

Task  ID 

Task  Name 

Sponsor 

Ql 

Q2 

A 

Q3 

Q4 

Q5 

Q6 

Q7 

Q8 

Main 

1 

Develop  model 

ONR 

Option 

2 

Develop  measures 

ASD  (C3I) 

' 

Option 

3 

Prototype  assessment  tool 

ASD  (C3I) 

Main 

4 

Design  TC2T  intervention 

ONR 

Main 

5 

Experimental  validation 

ONR 

Option 

6 

Visualization/assessment  tool 

ONR 

Option 

7 

TC2T  Training  tool 

ONR 

Briefings  to  the  clients 

Progress  reports  (quarterly) 
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Objectives 

•  A  fundamental  goal  of  military  is  to  ensure  that  C2  organizations  operate  decisively  and  synchronously  in  highly 
uncertain  and  dynamic  settings.  Individuals  succeed  in  these  settings  by  thinking  critically,  that  is  by  critiquing 
their  understanding  of  the  situation  at  hand,  refining  their  knowledge,  and  adapting  their  decision  making  and 
planning  to  the  problems  at  hand.  This  project  proposes  that  individual  team  members  collaborate  in  their 
application  of  critical  thinking  in  a  process  called  "team  critical  thinking”.  The  project  will  develop  the  concept  of 
collaborative  critical  thinking  within  C2  teams  from  three  research  threads  concerning:  individual  critical 
thinking,  team  process  and  architecture,  and  human  performance  in  information  age  warfare.  These  research 
threads  will  be  woven  together  to  create  a  theory,  validated  measures,  and  tools  and  techniques  that  help 
understand  and  support  team  critical  thinking.  The  team  collaboration  and  critical  thinking  theory  will  help 
explain  how  teams  critique  their  understanding  of  the  current  situation.  It  will  illustrate  how  teams 
incrementally  refine  their  shared  assessments  and  plans  or  radically  revise  their  beliefs  and  conceptual 
frameworks.  I  n  addition,  the  theory  will  help  explain  how  teams  turn  their  critical  faculties  on  themselves  to 
assess  and  adapt  the  fit  of  team  processes  and  team  structure  to  the  situation  at  hand. 

•  Based  on  these  measures,  training,  tools,  procedures  and  team  architectures  that  improve  team  critical  thinking 
will  be  developed.  The  end  product  will  be  a  solid  foundation  in  theory,  measurement,  and  practical  support  for 
improving  C2  teams  as  they  confront  the  challenging  and  varied  missions  of  the  21st  century. 

Research  Questions 

•  What  are  the  behavioral  markers  of  collaborative  critical  thinking? 

•  How  can  CCT  behaviors  and  their  effects  be  reliably  measured  in  a  semi-  or  fully  automated  fashion? 

•  Can  we  promote  CCT  behaviors  with  training  and  job  aids? 

Project  Status 

•  Two  theoretical  frameworks  have  been  developed,  one  concerning  collaboration  generally,  and  the  other 
addressing  the  role  of  CCT  within  collaboration. 

•  A  set  of  measures  has  been  drafted  that  addresses  several  aspects  of  collaboration  and  CCT. 

•  Design  discussions  are  underway  for  experiments,  tools,  and  training 
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